Papers with production pipelines
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (2025.acl-long)
Copied to clipboard
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, Iacopo Poli
| Challenge: | Encoder-only transformer models such as BERT offer a great performance-size tradeoff for retrieval and classification tasks compared to larger decoder models. |
| Approach: | They introduce a new transformer model, ModernBERT, which brings modern model optimizations to encoder-only transformer models. |
| Outcome: | The proposed model improves on the BERT transformer model and is faster and more memory efficient than the older models. |
CLOWER: A Pre-trained Language Model with Contrastive Learning over Word and Character Representations (2022.coling-1)
Copied to clipboard
Borun Chen, Hongyin Tang, Jiahao Bu, Kai Zhang, Jingang Wang, Qifan Wang, Hai-Tao Zheng, Wei Wu, Liqian Yu
| Challenge: | Pre-trained language models (PLMs) have achieved remarkable performance gains across numerous downstream tasks in natural language understanding. |
| Approach: | They propose a Chinese pre-trained language model that implicitly encodes words into characters . they propose 'contrastive learning over word' and 'character' representations to improve learning . |
| Outcome: | The proposed model can encode words into fine-grained representations without modification of production pipelines. |
Hire Me or Not? Examining Language Model’s Behavior with Occupation Attributes (2025.coling-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been widely integrated into production pipelines due to their impressive performance across multiple tasks. |
| Approach: | They construct a dataset using a standard occupation classification knowledge base and tested it on three families of LLMs. |
| Outcome: | The proposed framework analyzes LLMs’ behavior with respect to gender stereotypes in the context of occupation decision making. |